added ipfs dag import [path...] functionality too

still no tests yet, though it all works according to basic manual testing.

@achingbrain can I get some advice on this please? I tried to mirror the go-ipfs approach export and import are "commands" rather than "core" API features, so there's more logic in here that it doesn't get to defer to "core". But I think go-ipfs uses that command infrastructure for both CLI and server. But it looks like we have two separate systems for CLI and server and everything gets implemented in 3 places - cli & http-server, and http-client, and a lot of it defers to "core", so 4 or 5 places, with grpc probably doing another one?

Maybe I should be implementing this somewhere else and deferring the input and output streams somehow? import accepts either stdin or 1 or more files. export streams to stdout (but I wouldn't mind adding an --output which go-ipfs should have got). We probably need these things wired up through to server and even http client, but that leaves the question of where this functionality needs to live and how many places it needs to be wired up to and I'm not familiar enough with the intended architecture here.

I have 8 test fixtures totalling 2.9M (1.1M if I compress them) for this feature, but I'm noticing that ipfs-cli relies almost exclusively on mocks to do its testing, which is probably not ideal for these. So I'll need suggestions on where all that should go and examples for how to set things up for repetitive runs against reset block stores.

@rvagg typically you'd add functionality to ipfs-core and expose it in ipfs-cli and ipfs-http-server. I think we'd want to expose this over the HTTP API same as go-ipfs so we can test the http client against it, so it would need to go into core instead of being in the cli only.

The testing strategy - what sort of tests go where and how different components are tested - is documented here: https://github.com/ipfs/js-ipfs/blob/master/docs/DEVELOPMENT.md#testing-strategy

I'm not familiar enough with the intended architecture here

There's a small world of documentation here: https://github.com/ipfs/js-ipfs/tree/master/docs

I think go-ipfs uses that command infrastructure for both CLI and server

Yes, go-ipfs generates it's CLI and HTTP API from the same code whereas for better or for worse, js-ipfs has a more manual process.

I have 8 test fixtures totalling 2.9M (1.1M if I compress them) for this feature

Ouch, can these be generated on the fly by the tests?

@achingbrain the other dilemma I have here is the multiformats & ipld codec stack. @ipld/car uses multiformats and @ipld/dag-cbor itself already. I also need to do storage of raw bytes of blocks, plus traverse their decoded forms to gather the full DAG to export. The tools to get { cid, bytes, links } (as per this PR as it is now) from a block is easier with the new stack than it is with ipld which wants to turn a cid into a node, which is great for getting links but bytes gets discarded along the way so you have to go fishing for them elsewhere (or reimplement bits of ipld to do it).

So the choice is something like this:

1. Code against the existing ipld stack but suffer only the multiformats and @ipld/dag-cbor bundle expansion costs and have overly complicated code that will be awkward to undo if/when we get a full multiformats upgrade here.
2. Code against the new ipld stack and pull in the full set of codecs and suffer the bundle expansion costs of all of that (not massive, but not trivial), have slightly more straightforward code and be more future-proof if/when we get a full multiformats upgrade.

That might depend on timing for said upgrade so I'd appreciate your input on this.

Re test fixtures - I was wanting to import the ones that go-ipfs is using as they are, so we ensure both that the feature works properly and we have some level of parity with go-ipfs for it.

Status: dag export and dag import are implemented in core, cli, http-client and http-server. Minimal documentation and no tests. I've tested export manually in all of the places and it works like a charm. I've only tested import on the CLI, it's a bit advanced to set up a test with HTTP and I have run out of time this week.

Still needed: unit tests in cli, http-{client,server}, and some integration tests to match go-ipfs, plus it needs some additional docs. I'd really like to get t0054 from sharness working here via JS, it's 2.9M (1.1M compressed) of fixtures, but it's got really nice CAR coverage for various edge cases.

@rvagg where does the BlockCount property come from? It doesn't appear to be in the output of the dag.import http command?

Ah, I see - it's in an un-merged PR to go-ipfs: ipfs/kubo#8237

👌 looking really good, thanks @achingbrain
I really would like to see some test fixtures in here, though, it'd be good to figure out the best way to pull down large fixtures if it's going to be a problem for them to live in the repo. The ones @ribasushi put together for go-ipfs are pretty nice and cover the main edges of functionality for this feature.

Anecdotally for the fixture data the js implementation is 2-3.5x faster than go (ignoring test setup & teardown times):

$ npm run test:interface:http-go -- -t node

> ipfs@0.55.4 test:interface:http-go /Users/alex/Documents/Workspaces/ipfs/js-ipfs/packages/ipfs
> aegir test -f test/interface-http-go.js "-t" "node"

Test Node.js


  interface-ipfs-core over ipfs-http-client tests against go-ipfs
    .dag.export
      ✔ should export a car file (265ms)
      ✔ export of shuffled devnet export identical to canonical original (4412ms)
      ✔ export of shuffled testnet export identical to canonical original (32176ms)
    .dag.import
      ✔ should import a car file (488ms)
      ✔ should import a car file without pinning the roots (254ms)
      ✔ should import multiple car files (671ms)
      ✔ should import car with roots but no blocks (33776ms)
      ✔ should import lotus devnet genesis shuffled nulroot


  8 passing (1m)

vs

$ npm run test:interface:http-js -- -t node

> ipfs@0.55.4 test:interface:http-js /Users/alex/Documents/Workspaces/ipfs/js-ipfs/packages/ipfs
> aegir test -f test/interface-http-js.js "-t" "node"

Test Node.js


  interface-ipfs-core over ipfs-http-client tests against js-ipfs
    .dag.export
      ✔ should export a car file (136ms)
      ✔ export of shuffled devnet export identical to canonical original (1939ms)
      ✔ export of shuffled testnet export identical to canonical original (12438ms)
    .dag.import
      ✔ should import a car file (134ms)
      ✔ should import a car file without pinning the roots (99ms)
      ✔ should import multiple car files (197ms)
      ✔ should import car with roots but no blocks (14021ms)
      ✔ should import lotus devnet genesis shuffled nulroot


  8 passing (37s)

@achingbrain any idea what the difference might be? could it be on the rpc side, or is this only explainable by the core impl, and perhaps block store mechanism?

Would be interesting to try this same thing from the go http api, maybe.

The original perf of this in js was really bad because we were doing sequential block writes in datastore-fs and also not being very clever about how we walk a DAG when pinning recursively - I changed it to do parallel writes and to not walk child nodes that we've seen before during the traversal and it got much faster.

Would be interesting to try this same thing from the go http api, maybe

If I understand you correctly, this is what the test runs above are doing - using the go http api and the js http api to run the same tests.

well, 👌 , whatever's going on, this is pretty sweet, thanks again for picking it up!

🎉